Sqlite3 database is locked issue

Get Help
1

Good morning,

We’re experiencing a sqlite3 “database is locked” issue, only with the Job Composer app. Here is the error:

Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user username@domain.local with output: rake aborted!
ActiveRecord::StatementInvalid: SQLite3::BusyException: database is locked
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `initialize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `new'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `prepare'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:52:in `block (2 levels) in exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:49:in `block in exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:48:in `exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:107:in `query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:103:in `query_values'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/schema_statements.rb:35:in `data_sources'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/schema_cache.rb:239:in `prepare_data_sources'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/schema_cache.rb:89:in `data_source_exists?'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/model_schema.rb:380:in `table_exists?'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1108:in `get_all_versions'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1116:in `current_version'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1169:in `last_stored_environment'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/tasks/database_tasks.rb:60:in `check_protected_environments!'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:18:in `block (2 levels) in <top (required)>'

Caused by:
SQLite3::BusyException: database is locked
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `initialize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `new'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/sqlite3-1.4.2/lib/sqlite3/database.rb:147:in `prepare'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:52:in `block (2 levels) in exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:49:in `block in exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/sqlite3/database_statements.rb:48:in `exec_query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:107:in `query'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/database_statements.rb:103:in `query_values'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/abstract/schema_statements.rb:35:in `data_sources'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/schema_cache.rb:239:in `prepare_data_sources'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/connection_adapters/schema_cache.rb:89:in `data_source_exists?'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/model_schema.rb:380:in `table_exists?'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1108:in `get_all_versions'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1116:in `current_version'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/migration.rb:1169:in `last_stored_environment'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/tasks/database_tasks.rb:60:in `check_protected_environments!'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activerecord-6.1.7.6/lib/active_record/railties/databases.rake:18:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:setup => db:schema:load => db:check_protected_environments
(See full trace by running task with --trace)
'RAILS_ENV=production bin/rake db:setup' failed with output: [2024-05-16 10:20:06 -0500 ] DEBUG "\e[1m\e[35m (10182.6ms)\e[0m  \e[1m\e[34mSELECT sqlite_version(*)\e[0m"
Created database '/home/domain.local/username/ondemand/data/sys/myjobs/production.sqlite3'
[2024-05-16 10:20:22 -0500 ] DEBUG "\e[1m\e[35m (10199.2ms)\e[0m  \e[1m\e[34mSELECT sqlite_version(*)\e[0m"

And the stack trace:

/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:271:in `block (2 levels) in run_setup_production'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `chdir'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:257:in `block in run_setup_production'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `block in with_unbundled_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:692:in `with_env'
/usr/share/gems/gems/bundler-2.2.33/lib/bundler.rb:398:in `with_unbundled_env'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:254:in `run_setup_production'
/var/www/ood/apps/sys/dashboard/app/controllers/apps_controller.rb:35:in `show'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/lograge-0.12.0/lib/lograge/rails_ext/rack/logger.rb:18:in `call_app'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `block in call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/activesupport-6.1.7.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/opt/ood/ondemand/root/usr/share/gems/3.0/ondemand/3.0.3-1/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:107:in `process_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:419:in `block (3 levels) in start_threads'
/opt/rh/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

We have /home directories mounted via NFS on the ondemand node and these are the mount options that we are using:

nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=137.216.154.81,mountvers=3,mountport=597,mountproto=udp,local_lock=none

We did have another OnDemand instance with the local_lock=none option and we didn’t seem to encounter this issue. I’m probably missing something, but I didn’t know what to check next. Any suggestions are appreciated. I did try to delete and recreate the sqlite3 file in my /home directory, but this didn’t help.

2

Sorry for the trouble. Some googling on the errors for

Has turned up what looks to be an issue with SQLite3 in general handling cuncurrency that annoyingly happens only in production mode. There was a good writeup and some suggested fixes in this post below.

I would try to set that local_lock-=none on this instance and see if that makes a difference, as this issue looks to be generated by a confusion in a callback function used by SQLite to some C code but which leaves the ruby environment’s clock running which causes the timeout to hit for various reasons.

The short of it is that function won’t get called if there are no locks for the writes as it won’t need to check what to do if there is a lock in place. This might circumvent the logic which triggers that error.

3

Thanks for the suggestion! Unfortunately, we have the local_lock=none option on this instance, too, and we’re still having issues. I’m wondering if there is a way to disable the Job Composer option under the Jobs menu so it doesn’t show up? I don’t know if it really got all that much use anyway and if it isn’t working, we could remove it for now.

4

Searching for local_lock on this discourse provides a couple other instances in which this has happens.

I’d suggest maybe remounting the drives to see if they can be remounted with that option. I’d also suggest looking into the storage documentation to see if they even adhere to local_lock=none or nolock options. Different NFS providers do/allow different things, so it may be worth looking into the documentation for your NFS drives.

That said - yes you can disable the job composer. Simply chmod the /var/www/ood/apps/sys/myjobs/ directory to 700. Additionally you can chown the same directory to a different group (like some staff only Unix group) and set it to 750 so it’s only enabled for staff members.