Hi,
I built a simple Interactive App using batch_connect and it seems to work.
The only way I can terminate a job now is by the “Cancel” button. It would be nice to have a graceful shutdown option. Is it possible to add a button to the dashboard so that it is tied to some script (ideally on the compute node, but this is not critical)?
Alternatively, can I install a hook to the “Cancel” button handler, which is executed before the job is killed forcefully?
Thanks is advance
Hi and welcome!
You’ll have to remind me if this is a scheduler feature or not. In Slurm there are prologs, but they’re added by the administrators.
Which is to say - can you do this from the CLI as a regular unprivileged user? If yes, then we could likely add the feature to the UI. If you can’t do this from the CLI as a regular unprivileged user then there’s no way the web interface could as we run as that same regular unprivileged user using the command line interfaces for the scheduler.
Thanks for the reply Jeff!
I am not sure I understand your question. Our cluster runs PBS Professional.
I assume the “Cancel” button executes “qdel” under the hood. I am not aware of custom hooks before a job is killed.
The “application” I created is just a set of wrappers before.sh, script.sh, after.sh, clean.sh. They all are executed inside the job context on the compute node. The man job interacts via its own web interface. The scenario is similar to Jupyter, except that the program in question is much less flexible.
Let’s assume the program can perform an action (graceful shutdown) upon receiving SIGUSR1 signal. I am asking for a generic button, whose handler would be able to send such a signal. This handler should be run as the job owner. I can use ssh to the compute node if the handler is invoked on OOD server itself.
Ideally, the script would enjoy all the information available in the application scripts. The minimum is the JOBID.
Please let me know if my description is still unclear, thanks!
-Lev
Yea we don’t support anything like that currently. Mostly we like to use scheduler functionality. clean.sh actually does run after the main script - script.sh - exits. So if the user exits out of the application (say Jupyter or Rstudio), script.sh will complete and clean.sh gets executed at this time.
Lev: This is a very interesting idea. Unfortunately we have a backlog of hundreds of interesting ideas and very limited in-house developer time available to tackel them. We always welcome external contributors to our code base. I’d encourage you at a minimum to create an issue (or ideally some proof of concept code) via Contribute to Open OnDemand | Open OnDemand
@alanc The feature looks trivial, but unfortunately I can hardly contribute to its development. I speak C/Perl/Python but I have never been exposed to Ruby. I don’t understand how OOD is built and I found no examples of custom batch_connect buttons. Are there any?
Here’s a pull request we did recently to add a button to these cards, but honestly, LLMs I think are a great tool for this use case to bootstrap a modification with limited understanding of the code base. It won’t ultimately replace your need to know Ruby and Ruby on Rails, but could be a super useful starting point.