Hello - Not entirely OoD related but hoping someone can share some knowledge.
We have a user that could take advantage of submitting jobs via Slurm, the goal being that the job terminates once the job is complete, deleting the node. Currently they are using an interactive Matlab EC2 (this is AWS where time=$) where the EC2 will continue to stay up until the OoD job ends or is deleted.
The user is using an online license currently which seems to make this a bit more difficult.
Are you in control of the instance and how it boots up? (cloud-init type stuff?)
I’ve heard tell of folks who have daemons who check CPU usage. When they talk about it, these daemons will stop the job through Slurm. Seems like you could setup a daemon to kill the ec2 instance if it meets some condition (under 5% CPU usage for 1 hour for example).
As to OOD - I’m not sure how you’re able to use EC2 instances directly through OOD, I’m not familiar with that adapter but maybe you could limit these jobs to 1 or 2 hours only? And then offer 8 hours on Slurm? This will make an incentive to use Slurm instead.
That said - I’m not sure if I see a difference. The instance will be alive in both cases until the user acts or the job ends. So it’s not clear to me what the difference is unless you have no time limit on the EC2 adapter, in which case I’d say that’s the real issue. Again, I don’t know where this functionality is coming from, but it should have implemented time limits.
Thanks - I think we landed on having users add scancel $SLURM_JOBID to the end of their code which of course will terminate the job/EC2 which is the goal.
The environment is using OoD and AWS Parallel Cluster for those that are curious.