We recently ran into an issue with VNC Desktop sessions on our dedicated “VNC Login” host. We are using the LinuxHost adapter here and found that the /tmp/.X##-lock files were not getting cleaned up in all cases and we filled up all the display slots that were being searched. After clearing out the stale lock files I did a little testing and it looks like the lock files get cleaned up if I log out of the desktop, but if I cancel the “job” by hitting the delete button in the OOD session card the lock file will be left behind. I suspect that people are opting to delete sessions because the logout option is buried in our desktop setup. A side quest for me may be to make it easier to logout and then train our users to do that, but I would like to fix the root cause. This does not happen with jobs started by the scheduler (PBSPro in our case) so I believe that this problem is specific to the LinuxHost adapter probably due to the way that jobs are started and killed.
I’m wondering if anybody else has run into this issue and found a good way to fix it in the app definition? If this can’t be fixed in the app then I’m wondering if some changes could be made to the LinuxHost adapter to allow deleted jobs to clean up properly. I’m willing to dig a little deeper and work on a fix if somebody can point me in the right direction.
I am able to replicate the issue and can see the lock files persisting after the session has been killed. It looks like the ‘filled up all the display slots’ issue is happening because while we try to clean up lock files at the start of the session, each user can only clean up their own previous lockfiles, so many users on the same machine are bound to fill them up eventually, and then any users who don’t own a lockfile are unable to find a slot. A ‘real’ scheduler like PBS or slurm would handle creating tempdirs for each job and deleting them afterward, so it is expected behavior that vncserver never gets a clean exit.
Unfortunately there is no good entry point at the moment to attach additional cleanup scripts to job deletion, but there is one solution that might work. If you can get a script that runs as root and can delete lockfiles for sessions that are no longer active (regardless of owner), then you could run that with a crontab and periodically clean the stale files. That should be enough to (mostly) guarantee that an open spot will exist for each new user. The exact frequency would depend on how many users you have and how quickly it tends to fill up, but I would guess running it every 10 minutes or so would be a good starting point.
Finally, if the crontab solution doesn’t work, could you share your ood version and the content of job_script_content.sh and output.log from one of the failed desktop jobs? That would help narrow down exactly what is happening to cause the cleanup step to fail during job startup.
Thanks for looking into this and confirming that the problem isn’t specific to our site. I didn’t realize that there were steps during startup that re-claim the locks and sockets. This probably would have happened much sooner without those. I did confirm that if I generate a stale lock it does get re-used or re-created when I start a new session.
I can look at implementing a cron job to clean up stale files. We should be able to do that reliably, but it still feels like a hack to me. I’d like to see if we could get the files to clean up on exit. It looks like it should be possible to send a SIGTERM to the Xvnc process prior to killing the container process and that should trigger the cleanup. Do you think a patch to do that would be acceptable or am I missing something that would break if it were done that way?
Unfortunately there is not a great way to do that at the moment, as the LinuxHost adapter shares the structure of the ‘real scheduler’ adapters and does not have a way for an application-specific cleanup command before job deletion. I have opened Non-scheduler adapters need additional cleanup entrypoints · Issue #968 · OSC/ood_core to add this to LHA, but that would not be available until the next release. It is possible that once we get a PR for that ticket, you could override the updated classes in an initializer, but we will have to see what those changes actually look like to know how feasable that would be.
In the meantime, you could play around with overriding the stop_remote_session method of OodCore::Job::Adapters::LinuxHost::Launcher, which is where the container kill command gets executed. I’m not sure what the side effects might be, but if desktop sessions are the only application that has this issue, you could try issuing a SIGTERM to xvnc as part of every job deletion. That method currently looks like
def stop_remote_session(session_name, hostname)
cmd = ssh_cmd(hostname, ['/usr/bin/env', 'bash'])
kill_cmd = <<~SCRIPT
# Get the tmux pane PID for the target session
pane_pid=$(tmux list-panes -aF '\#{session_name} \#{pane_pid}' | grep '#{session_name}' | cut -f 2 -d ' ')
# Find the Singularity sinit PID child of the pane process
pane_sinit_pid=$(pstree -p -l "$pane_pid" | egrep -o 'sinit[(][[:digit:]]*|shim-init[(][[:digit:]]|appinit[(][[:digit:]]*' | grep -o '[[:digit:]]*')
# Kill sinit which stops both Singularity-based processes and the tmux session
kill "$pane_sinit_pid"
SCRIPT
call(*cmd, stdin: kill_cmd)
rescue Error => e
interpret_and_raise(e)
end