Job Composer breaking issue: database is locked for all (ood 3.1.7/rhel 8.10 update)

SOLVED!
Here’s how things shook out:

We received multiple other reports from researchers using different software packages that were complaining about file locks. rgas20’s post had clearly been suspecting some NFS issue, but we had not changed anything about our /home exports or mount options in a very long time.

Nonetheless, it occurred to me that if I could remove nfs from the equation, that would be smart. I was still perplexed that myjobs could touch but not “inflate” production.sqlite3 and wanted to see if that held true for local storage on, say, /tmp.

I hunted through this invaluable discourse forum and found exactly what I needed, a way to relocate “~/ondemand” (OOD_DATAROOT) for the dashboard and job-composer: Relocating /home/$user/ondemand folder
Once this was pointing at /tmp instead of /home, the Job Composer started working fine! Very interesting. I then redefined OOD_DATAROOT to point to /scratch which is also nfs, but exported from a different box.
…success again! Ah-HA.

As it turns out, on the system that exports /home, /var was 100% full. Clearing that situation up immediately allowed me to unset OOD_DATAROOT and use Job Composer like normal. No other write operations, anywhere, seemed to be impacted. Just this novel scattershot of file-lock reports. Hope this helps someone in the future!

1 Like